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DOCUMENT- IDENTIFIER: US 20020015166 Al 

TITLE: INFORMATION PROCESSING SYSTEM AND METHOD THEREFOR 



Application Filing Date (1) : 
19971229 

Brief Description of Drawings Paragraph (68) : 

[0094] FIG. 67 is a diagram showing an example where a schedule is extracted from 
electronic mail ; 

Brief Description of Drawings Paragraph (105) : 

[0131] FIG. 104 is a diagram showing an example wherein a controller receives a UI 
from a target model via a wireless LAN rather than directly, and displays it; 

Brief Description of Drawings Paragraph (106) : 

[0132] FIG. 105 is a flowchart showing the processing performed by a controller for 
the example wherein the controller receives a UI via a wireless LAN and displays it; 



Brief Description of Drawings Paragraph (107) : 

[0133] FIG. 106 is a flowchart showing the processing performed by a target model 
for the example wherein the controller receives a UI via a wireless LAN and displays 
it; 

Brief Description of Drawings Paragraph (108) : 

[0134] FIG. 107 is a flowchart showing the processing performed by a server for the 
example wherein the controller receives a UI via a wireless LAN and displays it; 

Brief Description of Drawings Paragraph (109) : 

[0135] FIG. 108 is a diagram showing an example wherein via a wireless LAN a 
controller receives a different UI for each user and displays it; 

Brief Description of Drawings Paragraph (110) : 

[0136] FIG. 109 is a flowchart showing the processing performed by a controller for 
the example wherein via a wireless LAN a controller receives a different UI for each 
user and displays it; 

Brief Description of Drawings Paragraph (111) : 

[0137] FIG. 110 is a flowchart showing the processing performed by a target model 
for the example wherein via a wireless LAN a controller receives a different UI for 
each user and displays it; 

Brief Description of Drawings Paragraph (112) : 

[0138] FIG. Ill is a flowchart showing the processing performed by a server for the 
example wherein via a wireless LAN a controller receives a different UI for each 
user and displays it; 

Detail Description Paragraph (26) : 

[0178] A request submitted by a user concerns paper sizes, printing quality, colors 
and time limit (e.g., completion of printing by 5 o'clock). The printer 
configuration covers the printer types and the number of printers in the system, and 
their performances or their current statuses (whether the printers are normally 
operated, whether there is a printing queue, or whether paper or toner is 
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insufficient) . As for a medium and a method used for notification, a notification is 
transmitted by voice through a user's terminal or by telephone, or a notification 
message is transmitted to a pager, or a document describing the contents of a 
notification is transmitted by electronic mail or by facsimile. 

Detail Description Paragraph (64) : 

[0216] When a PC 1102 acquires the status of a printer 1103 across a network and 
detects a change in the status, such as the occurrence of an error or an event, the 
PC 1102 determines a notification destination, such as a user or a manager, in 
consonance with the contents of the change, and employs a medium corresponding to 
the determined destination to transmit a notification. For example, to transmit a 
notification to a terminal 1101 of a user or a manager, the PC 1102 uses electronic 
mail , or a telephone 1104, or transmits a message to a pager 1105. 

Detail Description Paragraph (65) : 

[0217] Therefore, in the example in FIG. 10, for example, a notification of the 
reception of the document may be issued using electronic mail or a pager, instead of 
the telephone 707. 

Detail Description Paragraph (88) : 

[0240] Since, at step S157, the printer 104 can not execute the job, program control 
moves to step S160. Then, as at step S160 it is determined that the job for the 
output of information can be performed by the printer 103, at step S161 it is 
determined that the received information should be transmitted to the printer 103 . 
At step S162 the printer 104 sends an instruction to the printer 103 to print the 
information that is to be received (route 3) . At step S163 to notify the user that 
the instructed job was performed by the printer 103 electronic mail for the user is 
transmitted to the PC 101 (route 4) . 

Detail Description Paragraph (172) : 

[0324] When, at step S443, there is no printer available to which to transfer the 
job but there is another apparatus that can be employed by the user, such as a PC, a 
facsimile machine or a telephone, at step S444 it may be determined to perform the 
job by using one of the available apparatuses. Since the information can not be 
transmitted unchanged, at step S445 the information to be output to the printer is 
converted into an electronic mail document or a facsimile document, or is changed to 
voice information for the output. 

Detail Description Paragraph (238) : 

[0390] The above processing will now be described. In FIG. 62, at step S621 
information concerning a destination is acquired from information included in a 
print job. The information concerning the destination is obtained, as is explained 
in the twenty-ninth embodiment, either by interpreting the setup described in the 
print job, or by extracting it from information, such as E-mail , that is set 
separately from the contents of the print job. At this time, when, as in 
conventional use, a printer is employed as an output device for a personal computer, 
usually an addressee is not designated. When the printer is employed as a facsimile 
machine or for transmission of E-mail, as in the system in this embodiment, an 
addressee is normally designated. 

Detail Description Paragraph (239) : 

[0391] At step S622 a check is performed to determine whether or not an addressee is 
designated. At step S623 the addressee is notified of the arrival of a document. At 
step S624 the addresser information is acquired from the information included in the 
print job. The addresser information is acquired, as is explained in the 
twenty-ninth embodiment, either by interpreting the designation described in the 
print job, or by extracting it from information, such as E-mail, that is set 
separately from the print job. When, as in conventional use, a printer is employed 
as an output device for a common personal computer, normally the addresser falls 
within the control sphere of the system. However, when the printer is used as a 
facsimile machine, as in the system in this embodiment, usually the addresser is not 
within the control sphere. At step S625 a check is performed to determine whether 
the addresser is in the control sphere. At step S626 when the addressee is 
designated, a corresponding output tray is selected. When the addressee is not 
specified and only the addresser is specified, an output tray corresponding to the 
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addresser is selected. At step S627 the addresser or the addressee information is 
transmitted to the corresponding output tray, as is shown in FIG. 63A. Or, as is 
shown in FIG. 63B, the addresser or the addressee information and the output tray 
that is employed are displayed on the common display. When the user is set for each 
output tray, the individual trays are user dedicated trays. When the trays are not 
specifically assigned, the users of the output trays are changed as needed. 

Detail Description Paragraph (246) : 

[0398] The understanding unit 652 acknowledges, as needed, externally input 
information, such as keyboard input, voice input, E-mail or facsimile or news 
information, analyzes the received information and also the time concept contained 
in the words in sentences, and obtains an understanding of the contents of the 
sentences that are concerned with time. 

Detail Description Paragraph (247) : 

[0399] FIG. 67 is a diagram showing an example where schedule information is 
extracted fro m electronic mail . In the above process, specifically, as is shown in 
FIG. 67, the contents of a document received by E-mail are analyzed, and the time 
concepts such as "February 22nd M , "13:00" and "15:00" are found and analyzed. Then, 
an understanding of the sentence concerning the concept, "We have scheduled a patent 
system explanation meeting at conference room B from 13:00 to 15:00 on February 22" 
is obtained, and an action associated with the time can be correlated with the 
schedule . 

Detail Description Paragraph (249) : 

[0401] The understanding unit 652 analyses the contents of externally input 
information, such as keyboard input, voice input, E-mail , facsimile or news 
information. When the understanding unit 652 finds in the sentences a term (a word 
or a concept) that is unknown to the system, or a plan that the system has not yet 
executed, and when the word or the plan that is detected is not urgent, the 
understanding unit 652 studies the term or the plan by sequentially searching for 
information concerning it, and stores it as knowledge in the knowledge base 656. 

Detail Description Paragraph (254) : 

[0406] When, for example, it is ascertained, from information concerning an 
addresser, upon the receipt of the E-mail message shown in FIG. 67 that the mail is 
from an addresser who is not registered in the database, information concerning the 
addresser is extracted from the message and is registered in the personal database 
in the database 651. Further, when more information is obtained concerning the 
addresser who is not registered in the personal database, the information may be 
registered in the personal database. 

Detail Description Paragraph (257) : 

[0409] In the above explanation, the system prepares a reply to a user who sent the 
E-mail without confirming it with a user, and transmits the reply. This is because 
the sentence, "Please contact me as soon as possible if there is any difficulty" is 
in the message, and as a result of the analysis of this sentence, it is determined 
that a response is very urgently required. If the urgency of a response is 
determined to be low because a date for the holding the meeting is fairly advanced 
and the attendance at the meeting is arbitrary, the system confirms it with a user 
before transmitting a response. In other words, the system determines in which 
action to take in consonance with the degree of urgency. 

Detail Description Paragraph (261) : 

[0413] Input data can be data input via a keyboard, natural language information 
received by E-mail , documents or pictures input as images by a scanner, voice input 
via a microphone or images input by a camera. A character recognition process is 
preformed for a document that is read by the scanner, or a voice recognition process 
is performed for speech, so that input information in natural language can be 
obtained. 

Detail Description Paragraph (266) : 

[0418] In the example in FIG. 67, as a result of analysis of an E-mail document that 
is input, the following description concerning a person is acquired: 
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Detail Description Paragraph (281) : 

[0433] At step S680 E-mail is input and at step S681, of each word and sentence, the 
input E-mail document is analyzed. At step S683 it is ascertained from a signature 
and a header that "there is information concerning persons" and "an event, an 
explanation meeting, will be held". Further, it is ascertained that "a reply is 
requested if there is a difficulty" . 

Detail Description Paragraph (284) : 

[0436] In FIG. 69, at step S685 there is a plan to be executed: registration of the 
explanation meeting event in the schedule, and at step S686 a conflict is found in 
the schedule. Since there is a problem with the execution of a plan, at step S687 a 
reply to that effect is transmitted and a plan is made to notify a user that there 
is a problem. At step S685 there is a plan for writing a reply and transmitting it, 
and at step S686 no problem concerning the execution of the plan is found. At step 
S688, therefore, a reply indicating that the timing is inconvenient is prepared and 
transmitted to the addresser. At step S685 there is a plan for preparing a document 
for notification of a user. At step S686, there is no problem in execution, and at 
step S688 a response to a user is prepared describing that a reply was sent for 
reconsideration because of the contents of the E-mail and the conflict of the 
schedules. At step S685 there is no plan to be executed and at step S689 since there 
is an notice to a user, it is transmitted to the user. The processing is thereafter 
terminated. 

Detail Description Paragraph (316) : 

[0468] Information input by E-mail , by voice, via a keyboard, across the WWW, and by 
input devices, such as a telephone, a facsimile machine, a scanner and a camera, is 
analyzed by an input management unit 751 (step S760) . A core unit 752 ascertains the 
contents of the information and plans an appropriate process (step S761) . An output 
management unit 753 determines an output medium and prepares the contents to be 
output and an output route (step S763) . Then, the information is output by E-mail, 
by voice, across the WWW, or by output devices, such as a telephone, a facsimile 
machine, a printer and a copier. 

Detail Description Paragraph (323) : 

[0475] When, for example, E-mail is received that notifies the user of the holding 
of a conference, the input management unit 751 analyses the input information 
according to the procedures in FIG. 77. From the obtained result, the core unit 752 
prepares a plan, according to the procedures in FIG. 78, as to how the user and the 
system should handle the information. The plan is transmitted to the output 
management unit 753, which in turn performs the process according to the procedures 
in FIG. 79. When a response should be transmitted with the plan, a response is 
actually prepared and transmitted. 

Detail Description Paragraph (324) : 

[0476] Depending on the contents of the conference described in E-mail message and 
the status of the schedule of the user, at step S782 a plan is prepared for the 
aggressive transmission of the response to the user, so that preparation of the 
response is required. At step S791, therefore, it is determined that a response is 
to be transmitted. At step S792 the contents of the response are determined. When 
the schedules conflict and when it is uncertain which schedule should be selected or 
whether a user intends to attend a specific conference, the contents of a response 
asking for a decision are determined. At step S793 a medium is selected by which the 
contents of the response can be transmitted most effectively. When, for example, the 
user is outside the office, a medium, such as a telephone or a facsimile machine, is 
selected that can issue a notice to the user. When the user is being operating a 
personal computer, a medium, such as a personal computer, is selected that can 
effectively display various reference materials. At steps S794 and S795 a document 
and an image are prepared in consonance with the contents and the medium that are 
decided above, and a response is prepared using a method for converting the data 
into voiced natural language in consonance with a specific medium, and is issued. 

Detail Description Paragraph (357) : 

[0509] According to the system in this embodiment, without a user specifying an 
addressee, the addressee for a document is determined from bar code on a document 
that is read or information on a cover sheet, so that the document can be 
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transmitted to a correct addressee. 
Detail Description Paragraph (358) : 

[0510] As a result, appropriate information can be transmitted to a printer or a 
facsimile machine, by E-mail or via the WWW. 

Detail Description Paragraph (380) : 

[0532] At step S954 a process for inquiring the user (Dr. Aruna Rohra) is performed. 
In this case, as the result obtained at step S951 the user must meet John at the 
station, and an inquiry what to do is issued to the user. At step S955 what to do 
next is determined from the result of the inquiry, and whether or not an action is 
to be initiated is determined. In this example, the user instructs to ask Mr. Tanaka 
to meet him, the action is initiated according to the instruction. Since there is a 
request for Mr. Tanaka to act as a proxy, it is assumed that the action must be 
initiated. At step S956 a document for requesting Mr. Tanaka as a proxy is prepared 
and a contact is made with Mr. Tanaka. In this example, an electronic mail document 
is prepared and transmitted. In this example, information that John arrives on 28th, 
an instruction for meeting him for the user and the original document that caused 
the request are attached to automatically form the document. 

Detail Description Paragraph (386) : 

[0538] That is, (1) only when a controller faces an operating target device, it 
automatically recognizes the target device; (2) information for each device is 
acquired from the pertinent device via IrDA or via a wireless LAN; (3) a UI inherent 
to each device is provided from the information for the device; (4) an operation by 
voice can be performed; (5) common basic operation can be provided by using the same 
controller; (6) user identification information is employed to acquire necessary 
information for each user from the database or the address book and to implement the 
UI inherent to the user; and (7) the optimal UI is provided in consonance with the 
status . 

Detail Description Paragraph (419) : 

[0571] FIG. 104 is a diagram showing an example where UIs of devices are stored in a 
wireless LAN server 1041 and when a controller 960 faces a copier 962 as a target 
model and requests a UI, a UI is transmitted, upon a request from the copier 962, 
from the wireless LAN server 1041 to the controller 960, and is displayed. 

Detail Description Paragraph (426) : 

[0578] (5) The server 1041 transmits the UI of the target model, which is indicated 
by the target model data, via a wireless LAN to the controller 960, which is 
indicated by the controller identification data. 

Detail Description Paragraph (428) : 

[0580] FIG. 105 is a flowchart showing the processing performed by the controller 
960 when the controller receives a UI via the wireless LAN, instead of directly 
receiving it from a target model, and displays it. 

Detail Description Paragraph (430) : 

[0582] FIG. 106 is a flowchart showing the processing performed by the target model 
when the controller receives a UI via the wireless LAN, instead of directly 
receiving it from a target model, and displays it. The processing performed by the 
controller 960 will now be described. 

Detail Description Paragraph (432) : 

[0584] FIG. 107 is a flowchart showing the processing performed by the server 1041 
when the controller receives a UI via the wireless LAN, instead of directly 
receiving it from a target model, and displays it. The processing performed by the 
server 1041 will now be described. 

Detail Description Paragraph (435) : 

[0587] FIG. 108 is a diagram showing an example where UIs of devices are stored in a 
wireless LAN server 1041 and when a controller 960a or 960b faces a facsimile 
machine 963 as a target model and requests a UI, a different UI for each user is 
transmitted, upon a request from the facsimile machine 963, from the wireless LAN 
server 1041 to the controller 960a or 960b, and is displayed. 
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Detail Description Paragraph (442) : 

[0594] (5) The server 1041 transmits the UI of the target model, which is indicated 
by the user identification data and the target model data, via a wireless LAN to the 
controller 960a or 960b, which is indicated by the controller identification data. 
In the example in FIG. 108, a UI that is used for multi-address communication, which 
is difficult for beginners, is transmitted only to seniors, not to beginners. 

Detail Description Paragraph (446) : 

[0598] FIG. 109 is a flowchart showing the processing performed by the controller 
960 when the controller receives a different UI for each user, via the wireless LAN, 
instead of directly receiving it from a target model, and displays it. The 
processing performed by the controller 960 will now be described. 

Detail Description Paragraph (448) : 

[0600] FIG. 110 is a flowchart showing the processing performed by the target model 
when the controller receives a different UI for each user, via the wireless LAN, 
instead of directly receiving it from a target model, and displays it. The 
processing performed by the controller 960a or 960b will now be described. 

Detail Description Paragraph (450) : 

[0602] FIG. Ill is a flowchart showing the processing performed by the server 1041 
when the controller receives a different UI for each user via the wireless LAN, 
instead of directly receiving it from a target model, and displays it. FIG. 112 is a 
table showing UIs that are determined by the target model and user identification 
data. The processing performed by the server 1041 will now be described. 

Detail Description Paragraph (470) : 

[0622] (6) If a UI is not transmitted following a predetermined period of time after 
the UI request signal was transmitted, or if a UI is not transmitted during a period 
longer than a predetermined interval, which is set for transmission of UIs, another 
route is employed to transmit a UI request signal. In this example, when infrared 
communication can not be performed, communication through a wireless LAN is 
attempted. 

Detail Description Paragraph (473) : 

[0625] At step S1161 the controller 960 waits until an instruction from a user is 
input. At step S1162 a UI request signal for requesting a UI and controller 
identification data for identifying a controller are transmitted to the target model 
by an infrared ray communication method. At step S1163 a check is performed to 
determine whether or not the UI has been received. At step S1164 a check is 
performed to determine whether a predetermined period of time has elapsed following 
the transmission of the request. At step S1165 a UI request signal, for requesting a 
UI, and controller identification data, for identifying a controller, are 
transmitted by a route, such as across a wireless LAN, that is different from the 
route at step S1162. At step S1166 the received UI is displayed. 

Detail Description Paragraph (474) : 

[0626] In this case, the UI request signal that is directly transmitted to the 
target model is to be sent to the wireless LAN server 1041. However, since the 
wireless LAN server 1041 that receives the UI request signal is not sure of a target 
model, first a menu for selecting a model (function) is displayed. A UI 
corresponding to the selected model is displayed. Upon receipt of the model 
selection signal from the controller 960, the wireless LAN server 1041 may transmit 
to the controller 960 a UI that corresponds to the model, or may transmit the menu 
and a plurality of UIs available from the menu. 
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Drawing Description Text (68) : 

FIG. 67 is a diagram showing an example where a schedule is extracted from 
electronic mail ; 

Drawing Description Text (105) : 

FIG. 104 is a diagram showing an example wherein a controller receives a UI from a 
target model via a wireless LAN rather than directly, and displays it; 

Drawing Description Text (106) : 

FIG. 105 is a flowchart showing the processing performed by a controller for the 
example wherein the controller receives a UI via a wireless LAN and displays it; 

Drawing Description Text (107) : 

FIG. 106 is a flowchart showing the processing performed by a target model for the 
example wherein the controller receives a UI via a wireless LAN and displays it; 

Drawing Description Text (108) : 

FIG. 107 is a flowchart showing the processing performed by a server for the example 
wherein the controller receives a UI via a wireless LAN and displays it; 

Drawing Description Text (109) : 

FIG. 108 is a diagram showing an example wherein via a wireless LAN a controller 
receives a different UI for each user and displays it; 

Drawing Description Text (110) : 

FIG. 109 is a flowchart showing the processing performed by a controller for the 
example wherein via a wireless LAN a controller receives a different UI for each 
user and displays it; 

Drawing Description Text (111) : 

FIG. 110 is a flowchart showing the processing performed by a target model for the 
example wherein via a wireless LAN a controller receives a different UI for each 
user and displays it; 

Drawing Description Text (112) : 

FIG. Ill is a flowchart showing the processing performed by a server for the example 
wherein via a wireless LAN a controller receives a different UI for each user and 
displays it; 

Detailed Description Text (23) : 

A request submitted by a user concerns paper sizes, printing quality, colors and 
time limit (e.g., completion of printing by 5 o'clock). The printer configuration 
covers the printer types and the number of printers in the system, and their 
performances or their current statuses (whether the printers are normally operated, 
whether there is a printing queue, or whether paper or toner is insufficient) . As 
for a medium and a method used for notification, a notification is transmitted by 
voice through a user's terminal or by telephone, or a notification message is 
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transmitted to a pager, or a document describing the contents of a notification is 
transmitted by electronic mail or by facsimile. 

Detailed Description Text (48) : 

When a PC 1102 acquires the status of a printer 1103 across a network and detects a 
change in the status, such as the occurrence of an error or an event, the PC 1102 
determines a notification destination, such as a user or a manager, in consonance 
with the contents of the change, and employs a medium corresponding to the 
determined destination to transmit a notification. For example, to transmit a 
notification to a terminal 1101 of a user or a manager, the PC 1102 uses electronic 
mail, or a telephone 1104, or transmits a message to a pager 1105. 

Detailed Description Text (49) : 

Therefore, in the example in FIG. 10, for example, a notification of the reception 
of the document may be issued using electronic mail or a pager, instead of the 
telephone 7 07. 

Detailed Description Text (72) : 

Since, at step S157, the printer 104 can not execute the job, program control moves 
to step S160. Then, as at step S160 it is determined that the job for the output of 
information can be performed by the printer 103, at step S161 it is determined that 
the received information should be transmitted to the printer 103 . At step S162 the 
printer 104 sends an instruction to the printer 103 to print the information that is 
to be received (route 3) . At step S163 to notify the user that the instructed job. 
was performed by the printer 103 electronic mail for the user is transmitted to the 
PC 101 (route 4) . 

Detailed Description Text (152) : 

When, at step S443, there is no printer available to which to transfer the job but 
there is another apparatus that can be employed by the user, such as a PC, a 
facsimile machine or a telephone, at step S444 it may be determined to perform the 
job by using one of the available apparatuses. Since the information can not be 
transmitted unchanged, at step S445 the information to be output to the printer is 
converted into an electronic mail document or a facsimile document, or is changed to 
voice information for the output. 

Detailed Description Text (218) : 

The above processing will now be described. In FIG. 62, at step S621 information 
concerning a destination is acquired from information included in a print job. The 
information concerning the destination is obtained, as is explained in the 
twenty-ninth embodiment, either by interpreting the setup described in the print 
job, or by extracting it from information, such as E-mail, that is set separately 
from the contents of the print job. At this time, when, as in conventional use, a 
printer is employed as an output device for a personal computer, usually an 
addressee is not designated. When the printer is employed as a facsimile machine or 
for transmission of E-mail , as in the system in this embodiment, an addressee is 
normally designated. 

Detailed Description Text (219) : 

At step S622 a check is performed to determine whether or not an addressee is 
designated. At step S623 the addressee is notified of the arrival of a document. At 
step S624 the addresser information is acquired from the information included in the 
print job. The addresser information is acquired, as is explained in the 
twenty-ninth embodiment, either by interpreting the designation described in the 
print job, or by extracting it from information, such as E-mail, that is set 
separately from the print job. When, as in conventional use, a printer is employed 
as an output device for a common personal computer, normally the addresser falls 
within the control sphere of the system. However, when the printer is used as a 
facsimile machine, as in the system in this embodiment, usually the addresser is not 
within the control sphere. At step S625 a check is performed to determine whether 
the addresser is in the control sphere. At step S626 when the addressee is 
designated, a corresponding output tray is selected. When the addressee is not 
specified and only the addresser is specified, an output tray corresponding to the 
addresser is selected. At step S627 the addresser or the addressee information is 
transmitted to the corresponding output tray, as is shown in FIG. 63A. Or, as is 
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shown in FIG. 63B, the addresser or the addressee information and the output tray 
that is employed are displayed on the common display. When the user is set for each 
output tray, the individual trays are user dedicated trays. When the trays are not 
specifically assigned, the users of the output trays are changed as needed. 

Detailed Description Text (226) : 

The understanding unit 652 acknowledges, as needed, externally input information, 
such as keyboard input, voice input, E-mail or facsimile or news information, 
analyzes the received information and also the time concept contained in the words 
in sentences, and obtains an understanding of the contents of the sentences that are 
concerned with time. 

Detailed Description Text (227) : 

FIG. 67 is a diagram showing an example where schedule information is extracted from 
electronic mail . In the above process, specifically, as is shown in FIG. 67, the 
contents of a document received by E-mail are analyzed, and the time concepts such 
as "February 22nd" , "13:00" and "15:00" are found and analyzed. Then, an 
understanding of the sentence concerning the concept, "We have scheduled a patent 
system explanation meeting at conference room B from 13:00 to 15:00 on February 22" 
is obtained, and an action associated with the time can be correlated with the 
schedule . 

Detailed Description Text (229) : 

The understanding unit 652 analyses the contents of externally input information, 
such as keyboard input, voice input, E-mail, facsimile or news information. When the 
understanding unit 652 finds in the sentences a term (a word or a concept) that is 
unknown to the system, or a plan that the system has not yet executed, and when the 
word or the plan that is detected is not urgent, the understanding unit 652 studies 
the term or the plan by sequentially searching for information concerning it, and 
stores it as knowledge in the knowledge base 656. 

Detailed Description Text (233) : 

As is described above, in the system, the understanding unit 652 analyzes the 
contents of externally input information and also analyzes the object. The planning 
unit 653 prepares a plan of an action for the system to take to achieve the object. 
The execution unit 654 performs the processing. And the response unit 655 notifies a 
user of the results of the processing. When, for example, it is ascertained, from 
information concerning an addresser, upon the receipt of the E-mail message shown in 
FIG. 67 that the mail is from an addresser who is not registered in the database, 
information concerning the addresser is extracted from the message and is registered 
in the personal database in the database 651. Further, when more information is 
obtained concerning the addresser who is not registered in the personal database, 
the information may be registered in the personal database. 

Detailed Description Text (236) : 

In the above explanation, the system prepares a reply to a user who sent the E-mail 
without confirming it with a user, and transmits the reply. This is because the 
sentence, "Please contact me as soon as possible if there is any difficulty" is in 
the message, and as a result of the analysis of this sentence, it is determined that 
a response is very urgently required. If the urgency of a response is determined to 
be low because a date for the holding the meeting is fairly advanced and the 
attendance at the meeting is arbitrary, the system confirms it with a user before 
transmitting a response. In other words, the system determines in which action to 
take in consonance with the degree of urgency. 

Detailed Description Text (240) : 

Input data can be data input via a keyboard, natural language information received 
by E-mail, documents or pictures input as images by a scanner, voice input via a 
microphone or images input by a camera. A character recognition process is preformed 
for a document that is read by the scanner, or a voice recognition process is 
performed for speech, so that input information in natural language can be obtained. 



Detailed Description Text (245) : 

In the example in FIG. 67, as a result of analysis of an E-mail document that is 
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input, the following description concerning a person is acquired: To: 
toshima@abc . canon. co. jp, rohra@abc, canon .co.jp, kazuyo@abc . canon. co.jp From 
ichiro@abc.canon.co.jp I am Suzuki of system promotion section. Suzuki 
(ichiro@abc.canon.co.jp) 044-123-4569 (ext. 654-3210) Canon Inc. Intellectual 
Property Head Office System Promotion Section Assuming that a person whose mail 
address is "kazuyo@abc . canon . co . jp M is already registered in the personal database 
and the other persons are not registered, as is shown in FIG. 67, data for PERSON 1, 
2 and 3 are extracted and registered in the database 651. 

Detailed Description Text (250) : 

At step S680 E-mail is input and at step S681, of each word and sentence, the input 
E-mail document is analyzed. At step S683 it is ascertained from a signature and a 
header that "there is information concerning persons" and "an event, an explanation 
meeting, will be held". Further, it is ascertained that "a reply is requested if 
there is a difficulty". 

Detailed Description Text (253) : 

In FIG. 69, at step S685 there is a plan to be executed: registration of the 
explanation meeting event in the schedule, and at step S68 6 a conflict is found in 
the schedule. Since there is a problem with the execution of a plan, at step S687 a 
reply to that effect is transmitted and a plan is made to notify a user that there 
is a problem. At step S685 there is a plan for writing a reply and transmitting it, 
and at step S686 no problem concerning the execution of the plan is found. At step 
S688, therefore, a reply indicating that the timing is inconvenient is prepared and 
transmitted to the addresser. At step S685 there is a plan for preparing a document 
for notification of a user. At step S686, there is no problem in execution, and at 
step S688 a response to a user is prepared describing that a reply was sent for 
re-consideration because of the contents of the E-mail and the conflict of the 
schedules. At step S685 there is no plan to be executed and at step S689 since there 
is an notice to a user, it is transmitted to the user. The processing is thereafter 
terminated. 

Detailed Description Text (284) : 

Information input by E-mail , by voice, via a keyboard, across the WWW, and by input 
devices, such as a telephone, a facsimile machine, a scanner and a camera, is 
analyzed by an input management unit 751 (step S760) . A core unit 752 ascertains the 
contents of the information and plans an appropriate process (step S761) . An output 
management unit 753 determines an output medium and prepares the contents to be 
output and an output route (step S763) . Then, the information is output by E-mail, 
by voice, across the WWW, or by output devices, such as a telephone, a facsimile 
machine, a printer and a copier. 

Detailed Description Text (291) : 

When, for example, E-mail is received that notifies the user of the holding of a 
conference, the input management unit 751 analyses the input information according 
to the procedures in FIG. 77. From the obtained result, the core unit 752 prepares a 
plan, according to the procedures in FIG. 78, as to how the user and the system 
should handle the information. The plan is transmitted to the output management unit 
753, which in turn performs the process according to the procedures in FIG . 79. When 
a response should be transmitted with the plan, a response is actually prepared and 
transmitted. 

Detailed Description Text (292) : 

Depending on the contents of the conference described in E-mail message and the 
status of the schedule of the user, at step S782 a plan is prepared for the 
aggressive transmission of the response to the user, so that preparation of the 
response is required. At step S791, therefore, it is determined that a response is 
to be transmitted. At step S792 the t contents of the response are determined. When 
the schedules conflict and when it is uncertain which schedule should be selected or 
whether a user intends to attend a specific conference, the contents of a response 
asking for a decision are determined. At step S793 a medium is selected by which the 
contents of the response can be transmitted most effectively. When, for example, the 
user is outside the office, a medium, such as a telephone or a facsimile machine, is 
selected that can issue a notice to the user. When the user is being operating a 
personal computer, a medium, such as a personal computer, is selected that can 
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effectively display various reference materials. At steps S794 and S795 a document 
and an image are prepared in consonance with the contents and the medium that are 
decided above, and a response is prepared using a method for converting the data 
into voiced natural language in consonance with a specific medium, and is issued. 

Detailed Description Text (315) : 

According to the system in this embodiment, without a user specifying an addressee, 
the addressee for a document is determined from bar code on a document that is read 
or information on a cover sheet, so that the document can be transmitted to a 
correct addressee. 

Detailed Description Text (316) : 

As a result, appropriate information can be transmitted to a printer or a facsimile 
machine, by E-mail or via the WWW. 

Detailed Description Text (338) : 

At step S954 a process for inquiring the user (Dr. Aruna Rohra) is performed. In 
this case, as the result obtained at step S951 the user must meet John at the 
station, and an inquiry what to do is issued to the user. At step S955 what to do 
next is determined from the result of the inquiry, and whether or not an action is 
to be initiated is determined. In this example, the user instructs to ask Mr. Tanaka 
to meet him, the action is initiated according to the instruction. Since there is a 
request for Mr. Tanaka to act as a proxy, it is assumed that the action must be 
initiated. At step S956 a document for requesting Mr. Tanaka as a proxy is prepared 
and a contact is made with Mr. Tanaka. In this example, an electronic mail document 
is prepared and transmitted. In this example, information that John arrives on 28th, 
an instruction for meeting him for the user and the original document that caused 
the request are attached to automatically form the document. 

Detailed Description Text (344) : 

That is, (1) only when a controller faces an operating target device, it 
automatically recognizes the target device; (2) information for each device is 
acquired from the pertinent device via IrDA or via a wireless LAN; (3) a UI inherent 
to each device is provided from the information for the device; (4) an operation by 
voice can be performed; (5) common basic operation can be provided by using the same 
controller; (6) user identification information is employed to acquire necessary 
information for each user from the database or the address book and to implement the 
UI inherent to the user; and (7) the optimal UI is provided in consonance with the 
status . 

Detailed Description Text (367) : 

FIG. 104 is a diagram showing an example where UIs of devices are stored in a 
wireless LAN server 1041 and when a controller 960 faces a copier 962 as a target 
model and requests a UI, a UI is transmitted, upon a request from the copier 962, 
from the wireless LAN server 1041 to the controller 960, and is displayed. 

Detailed Description Text (369) : 

The communication between the controller and the target model will now be described. 

(1) A user directs the controller 960 to a target model (a copier 962) . (2) The user 
touches a touch panel on the controller 960 in order to determine the timing at 
which the controller 960 transmits a UI request signal to the target model. Or, the 
controller 960 is set in advance to transmit a UI request signal at a predetermined 
interval. (3) The controller 960 transmits controller identification data 

(controller A) to identify itself and a UI request signal to the target model. (4) 
When the target model receives the controller identification data and the UI request 
signal from the controller, the target model transmits, to the server 1041 that 
manages the target model, the controller identification data (controller A) and 
target model identification data (copier) to identify itself, and requests the 
transmission of the UI . (5) The server 1041 transmits the UI of the target model, 
which is indicated by the target model data, via a wireless LAN to the controller 
960, which is indicated by the controller identification data. (6) The controller 
960 receives the UI addressed to itself, and displays the UI . 



Detailed Description Text (370) : 

FIG. 105 is a flowchart showing the processing performed by the controller 960 when 
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described. It should be noted that (2) and (3) are not necessarily required. When, 
for example, in (4) a target model generates a signal to the controller at 
predetermined intervals, the same effect can be obtained. In other words, only the 
essential portion is shown in FIG. 115. (1) A user directs the controller 960 to a 
target model. (2) The user touches a touch panel on the controller 960 in order to 
determine the timing at which the controller 960 transmits a UI request signal to 
the target model. Or, the controller 960 is set in advance to transmit a UI request 
signal at a predetermined interval. (3) The controller 960 transmits a UI request 
signal to the target model. (4) A UI signal is transmitted upon the receipt of the 
UI request signal from the controller 960, or according to the setup that a UI 
request signal at a specified interval. (5) The controller 960 that receives the UI 
signal displays the UI . (6) If a UI is not transmitted following a predetermined 
period of time after the UI request signal was transmitted, or if a UI is not 
transmitted during a period longer than a predetermined interval, which is set for 
transmission of UIs, another route is employed to transmit a UI request signal. In 
this example, when infrared communication can not be performed, communication 
through a wireless LAN is attempted. (7) The UI acquired across the LAN is 
displayed. 

Detailed Description Text (395) : 

At step S1161 the controller 960 waits until an instruction from a user is input. At 
step S1162 a UI request signal for requesting a UI and controller identification 
data for identifying a controller are transmitted to the target model by an infrared 
ray communication method. At step S1163 a check is performed to determine whether or 
not the UI has been received. At step S1164 a check is performed to determine 
whether a predetermined period of time has elapsed following the transmission of the 
request. At step S1165 a UI request signal, for requesting a UI, and controller 
identification data, for identifying a controller, are transmitted by a route, such 
as across a wireless LAN, that is different from the route at step S1162. At step 
S1166 the received UI is displayed. 

Detailed Description Text (396) : 

In this case, the UI request signal that is directly transmitted to the target model 
is to be sent to the wireless LAN server 1041. However, since the wireless LAN 
server 1041 that receives the UI request signal is not sure of a target model, first 
a menu for selecting a model (function) is displayed. A UI corresponding to the 
selected model is displayed. Upon receipt of the model selection signal from the 
controller 960, the wireless LAN server 1041 may transmit to the controller 960 a UI 
that corresponds to the model, or may transmit the menu and a plurality of UIs 
available from the menu. 

Detailed Description Text (485) : 

The filing server 122, which serves as the information processing apparatus 412 
connected via the network, receives the instruction task from the reception unit 43, 
analyzes the received instruction, and performs planning on the data. The execution 
unit 47 then transmits corresponding information 125 "abc.doc" to John by E-mail . 

Detailed Description Text (550) : 

Similarly, further necessary actions are storage, filing and transferring. Any type 
of information can be extracted; a document, a file, electronic mail, a schedule, a 
status report, a log, voice mail or telephone call. 

Detailed Description Text (558) : 

FIG. 14 8 is a diagram showing a further example pull session. Information type is 
electronic mail and a requested format is voice. The procedures are, however, the 
same as previously described. 

Detailed Description Text (574) : 

FIG. 154 is a diagram showing another example push task, which is a transmission of 
electronic mail . An apparatus A (printer) at Kosugi is to transmit electronic mail 
"Get design from Bill" to an apparatus B (having a different network sub-domain from 
John's PC and the apparatus A) in New York. 



Detailed Description Text (596) : 

Upon receipt of the information, the apparatus B performs the required processing in 
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consonance with the information, decides an optimal transmission method, such as 
E-mail , and transmits the document to John as is instructed. 
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the controller receives a UI via the wireless LAN, instead of directly receiving it 
from a target model, and displays it. 

Detailed Description Text (372) : 

FIG. 106 is a flowchart showing the processing performed by the target model when 
the controller receives a UI via the wireless LAM, instead of directly receiving it 
from a target model, and displays it. The processing performed by the controller 960 
will now be described. 

Detailed Description Text (374) : 

FIG. 107 is a flowchart showing the processing performed by the server 1041 when the 
controller receives a UI via the wireless LAN, instead of directly receiving it from 
a target model, and displays it. The processing performed by the server 1041 will 
now be described. 

Detailed Description Text (377) : 

FIG. 10 8 is a diagram showing an example where UIs of devices are stored in a 
wireless LAN server 1041 and when a controller 960a or 960b faces a facsimile 
machine 963 as a target model and requests a UI, a different UI for each user is 
transmitted, upon a request from the facsimile machine 963, from the wireless LAN 
server 1041 to the controller 960a or 960b, and is displayed. 

Detailed Description Text (379) : 

The communication between the user, the controller 960a or 960b and the target model 
will now be described. (1) A user directs the controller 960a or 960b to a target 
model. (2) The user touches a touch panel on the controller 960a or 960b in order to 
determine the timing at which the controller 960a or 960b transmits a UI request 
signal to the target model. Or, the controller 960a or 960b is set in advance to 
transmit a UI request signal at a predetermined interval. (3) The controller 960a or 
960b transmits controller identification data (controller A or controller B) to a 
identify itself, user identification data (a senior or a beginner) for identifying 
the user, and a UI request signal to the target model. (4) When the target model 
receives, from the controller 960a or 960b, the controller identification data, the 
user identification data and the UI request signal, the target model transmits, to 
the server 1041 that manages the target model, the received controller 
identification data, the user identification data and the target model 
identification data to identify itself. (5) The server 1041 transmits the UI of the 
target model, which is indicated by the user identification data and the target 
model data, via a wireless LAN to the controller 960a or 960b, which is indicated by 
the controller identification data. In the example in FIG. 108, a UI that is used 
for multi-address communication, which is difficult for beginners, is transmitted 
only to seniors, not to beginners. (6) The controller 960a or 960b receives the UI 
addressed to itself, and displays the UI . 

Detailed Description Text (382) : 

FIG. 109 is a flowchart showing the processing performed by the controller 960 when 
the controller receives a different UI for each user, via the wireless LAN, instead 
of directly receiving it from a target model, and displays it. The processing 
performed by the controller 960 will now be described. 

Detailed Description Text (384) : 

FIG. 110 is a flowchart showing the processing performed by the target model when 
the controller receives a different UI for each user, via the wireless LAN, instead 
of directly receiving it from a target model, and displays it. The processing 
performed by the controller 960a or 960b will now be described. 

Detailed Description Text (386) : 

FIG. Ill is a flowchart showing the processing performed by the server 1041 when the 
controller receives a different UI for 'each user via the wireless LAN, instead of 
directly receiving it from a target model, and displays it. FIG. 112 is a table 
showing UIs that are determined by the target model and user identification data. 
The processing performed by the server 1041 will now be described. 

Detailed Description Text (393) : 

The communication between the user, the controller and the target model will now be 



